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Sustainment  Investment:  the  Problem 

How  do  we  make  decisions  about  the  allocation  of  funds 
between  areas? 


What  are  the  consequences  of  poor  and  delayed  decisions? 


How  can  we  forecast  the  consequences  of  various  alternatives? 
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Agenda 


Sustainment  Overview 
Understanding  Our  Simulation 
Using  Scenarios  to  Apply  the  Tool 
Results  and  Challenges 
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Demand  for  Sustainment  is  “Lumpy” 

Slide  from  PSM  Conference,  2012 


Program  Software  Maintenance  Release  Profile 
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The  Decision  Trade-Space 

Context 

We  are  managing  the  sustainment  of  a  system. 

•  Some  modernization  work  requiring  a  new  technology  is  funded. 

•  New  threats  introduce  new  mission  performance  criteria. 

•  Changes  to  interfacing  system  obsoletes  a  component. 

Potential  Consequences  of  a  Technology  Change 

•  Productivity  estimates  may  change  requiring  cost  and  schedule  adjustments. 

•  Failure  to  update  process  may  result  in  product  quality  deficiency. 

Trade  Space 

Budget  allocation  decision  between  product  development  costs  vis-a-vis 
the  cost  of  improving  the  process  and  training  people. 

Problem 

Multiple  stakeholders  causes  decisions  delays. 
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The  Sustainment  Business 


The  process  is  inside  the  gray  box. 

Customers  are  outside.  Work  input  translates  “demand”  to  requirements. 


Demands  that  involve  new  technology  require  re-skilling  and  retooling. 


Sustainment  Process 


Demand  )  Req. Accepted 
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change 
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Outcome 


•  Readiness 

•  Availability 
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reports 


While  the  work  inside  the  gray  box  is  often  measured, 
“demand”  and  “outcome”  can  and  should  be  measured  also. 
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Stakeholders  in  Sustainment 

Funding  decisions  may  affect  multiple  criteria 


Performance  Goals 


Agenda 


Sustainment  Overview 

Understanding  Our  Simulation 

Using  Scenarios  to  Apply  the  Tool 
Results  and  Challenges 
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Reasons  to  Use  A  Simulation 


Type  of  problem 

•  Multiple  sources  of  change  (stakeholders,  technology,  threat) 

•  Change  events  occur  asynchronously 

•  Multiple  opportunities  for  decisions  to  control  the  situation 

•  Outcomes  of  changes  and  decisions  do  not  appear  at  the  same  time  and  appear 
after  significant  delays. 

The  Investment  Simulation  Model 

•  Clarifies  the  trade  space  for  the  decisions. 

•  “Slider  bars”  represent  alternative  decisions;  sensitivity  can  be  tested. 

•  Simulation  provides  insight  about  future  effects  of  decisions  and 

•  Suggests  how  to  prevent  problems  before  they  become  too  expensive. 

Next:  A  sample  piece  of  the  simulation  model. 
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Mission  Operations,  Requesting 
Enhancements 


Beginning  with  Mission  Performance  Goals: 

These  goals  normally  expand  overtime.  If  we  keep  up,  they  keep  expanding.  (Good) 
Changing  technology  will  result  in  additional  requests. 

As  requests  are  received,  the  sustainment  organization  has  to  decide  which  to  accept 
and  how  to  deliver  the  enhanced  product. 

The  request  can  be  “sized”  before  being  accepted  for  implementation. 
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Sustainment  Capacity  Implements  Mission 


Investing  in  People,  Processes  and  Tools 


“Capability”  used  as  a 
proxy  for  peoples’  skills, 
processes,  and  tools. 

“Training”  is  a  proxy  for 
investment  since  “the 
people  are  the  process.” 

Sustainment  capacity 

=  Capability  *  Staff 
Avail. 


train  or  tool 
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Funding 

The  Program  Office  releases  (budgeted  funds)  to  the  sustainers.  The 
rate  of  release  affects  the  sustainers  ability  to  deliver 


The  program  office  may 
allocate  the  budget. 

Not  all  the  funds  come  at  one 
time  so  the  stock  is  used  to 
model  effects  of  delay. 


budget 
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All  the  Model  Parts,  Together 


Mission  Capability 
Requests 


_  mission  performance 


Training  & 
Process 


Strategy 


Funding 
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Simulation  Interface:  Slider  bars  control  behavior 
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Precision  and  Accuracy 


Model  Precision  and  Accuracy 

•  Simulations  are  often  not  very  precise.  We  may  not  be  able  to  tell 
whether  something  will  cost  $10  or  $12  with  this  type  model. 

•  A  complicated  model  like  this  requires  some  formal  investigation  to 
determine  whether  it  properly  and  accurately  represents  real  system 
behavior. 

•  Once  we  have  confidence  in  the  behavior,  we  can  evaluate  whether 
we  want  more  data  (at  a  cost)  to  improve  the  precision. 

How  do  we  validate  system  behavior? 

We  use  scenarios  and  prior  history. 
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Agenda 


Sustainment  Overview 
Understanding  Our  Simulation 

Using  Scenarios  to  Apply  the  Tool 

Results  and  Challenges 
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Writing  a  Scenario  in  4  Parts 

1 .  Context 

Describe  what  is  happening  today  (regular  sustainment,  limitations  on 
development  staffing,  limitations  on  decision  space?). 

2.  Stimulus 

An  external  event  necessitates  a  decision  to  reassign  staff,  adjust  schedule 
priorities  or  invest  in  new  people-skills  and  processes. 

3.  Responses 

How  would  we  reallocate  funding/staff?  How  can  priorities  be  adjusted? 

4.  Outcome 

How  does  the  system  respond  to  our  decisions? 

•  Do  we  lose  productivity?  Staff?  Harder  to  train? 

•  Do  we  lose  customers?  Sponsorship? 

•  Are  we  taking  longer  to  deliver? 

•  How  did  the  various  stocks  and  flows  perform  and  does  this  behavior  match  history? 
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Sequestration  Scenario:  Evaluate  Effects 

Context 

Normal  sustainment  work  at  a  weapons  test  facility. 

Stimulus 

Congress  decides  that  Federal  workers  should  work  only  4  days/week  to  reduce 
costs  by  20%. 

Responses 

Test  effects  of  staff  furlough  with  and  without  continued  training. 

Outcome 

Depends  on  duration  and  number  of  personnel  affected. 

•  If  sequestration  lasts  for  12  months  and  100  people  are  affected  then  120  FTE 
(months)  personnel  is  the  total  effect  on  productivity. 

•  Avoiding  training  adds  back  total  of  12  FTE  months  (approx.  5%).  Attrition  may 
increase  effects  because  “no  training  and  lower  pay”  is  a  demotivator. 
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New  Threat,  No  Budget 

Context 

Operational  capability  is  currently  in  use  in  the  theater. 

Stimulus 

Mission  effectiveness  is  challenged  by  a  new  threat. 

•  The  threat  warrants  an  upgrade  to  the  system  capability.  The  cost  of  the  upgrade  is 
approximately  $100M  if  performed  by  the  sustainment  organization  and  is  estimated  to  take 
12  months  to  deliver.  The  budget  is  limited. 

Responses 

Add  feature  to  current  release  or  wait  to  add  features  when  budget  increases? 

Outcome 

•  Adding  feature  delays  current  release  by  24  months  even  with  additional  funding. 

•  Delaying  additional  feature  until  next  release  takes  only  slightly  more  than  24  months  and 
results  in  better  performance  for  both  sustainers  and  warfighter. 


Software  Engineering  Institute  Carnegie  Mellon  University 


Modeling  Sustainment  Investment 
May  2015 

©2015  Carnegie  Mellon  University 


22 


Agenda 


Sustainment  Overview 
System  Dynamics  (SD)  Modeling 
Understanding  Our  SD  Model 
Using  Scenarios  to  Apply  the  Tool 

Results  and  Challenges 
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Value  of  Using  the  Simulation 

From  the  model,  we  can  see  certain  cause-and-effect  relationships. 

•  If  we  are  requested  to  develop  a  capability  based  on  new  technology, 

then  we  must  develop  internal  capabilities  to  meet  that  demand, 
else  we  will  not  achieve  the  same  level  of  sustainment  performance  and 
will  require  longer  schedules  to  perform  block  releases. 

•  Longer  schedules  will  also  increase  costs. 

We  use  the  model  to  show  others  that 

•  re-tooling  the  organization  saves  more  time  and  money  than 

•  simply  working  harder  and  OJT  to  develop  internal  capabilities. 

Using  the  model  lets  us  show  how  the  negative  consequences  might 
affect  development  schedule  and  mission  performance. 

Delivery  Schedules  and  Mission  Performance  are  more  important 
than  costs  to  mission  leaders  who  may  ignore  requests  for  funding. 
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Challenges 

Analyzing  the  3  “gaps” 

•  Each  gap  compares  performance  to  a  goal.  The  performers  measure  their 
performance  for  reasons  differently  from  the  goal-setters.  Since  the  measures 
don’t  match,  there  must  be  a  discussion  of  the  meaning  of  any  gap. 

Calibration 

•  Calibration  is  always  imperfect  because  of  abstraction,  but  there  will  be  reasons 
to  make  it  better.  What  is  the  “economic  value  of  the  information?” 

•  As  decision-making  processes  are  changed,  the  model  will  need  updates. 

Validation 

•  Historical  data  and  improvements  in  the  monitoring  of  system  behavior  will 
provide  supporting  information  for  scenarios. 

•  Better  model  results  will  come  from  comparing  “demand”  and  “outcome”  data 
than  can  be  obtained  by  looking  at  the  process  from  the  inside  of  the  box  alone. 
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Summary 

Model  behavior  appears  to  match  fairly  well  to  the  available  information. 

The  amount  of  data  required  so  far  has  been  reasonable. 

•  We  have  been  able  to  use  existing  sources.  We  have  not  examined  all  the 
available  data. 

The  concept  of  measuring  “capabilities”  is  not  sufficiently  addressed. 

•  Sizing  or  counting  capabilities  for  both  customer  demand  and  sustainment 
capability  (processes  and  people)  needs  some  work.  The  current  proxies  might 
not  be  sufficiently  accurate. 

The  Vensim™  (modeling  tool)  capability  makes  it  fairly  easy  to  test 

several  possible  allocation  decisions  very  quickly. 

•  If  we  can  reasonably  validate  the  model  so  that  managers  have  confidence  in 
system  behavior,  then  it  will  empower  managers  to  make  stronger 
recommendations  about  allocating  funds  and  accepting  work. 
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Stocks  and  Flows:  The  Building  Blocks 

Stocks  hold  things  and  Flows  with  Valves  fill  or  empty  stocks. 

Arrows  are  inputs  to  calculations  and  valves. 

A  simple  model  with  positive  feedback  causes  an  “exponential  growth.” 


A  $100  initial  value  for  savings,  with  a  positive  interest  rate,  exhibits  the 
effects  of  compound  interest. 

The  arrows  show  Savings$$  and  InterestRate  are  used  to  calculate  input 
to  savings.  The  Stock  acts  as  an  “integration  engine”. 

Check  units  of  measure  to  make  sure  equations  work.. 
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Calibration  Questions 


How  would  you  size  a  new  mission  capability  request? 

How  would  you  size  sustainment  capability  requests? 

Mission  Capability  Size  Requires  Knowledge  of  System 

•  Use  {common,  uncommon,  innovative}  as  words  to  classify  the 
capability  request. 

•  Partition  the  capability  into  requirements  at  the  component  (CSCI)  level 
and  do  the  same  thing  again.  (Use  “new  AESA  radar”) 

•  Use  {common=1 ,  uncommon=3,  innovative=9}  for  size  of  each 
component. 

•  Use  SUM(all  requirement  in  the  enhancement)  to  get  the  relative  size. 

•  This  relative  sizing  will  have  a  consistent  correlation  to  effort.  Using  any 
analogy  or  size  estimate  with  this  basis  will  be  effective. 

People/Process  Capability  Size 

•  Can  be  estimated  by  draining  days  per  capability  per  person. 
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Assessing  Gaps 


Gap  Definition 

•  The  performance  goal  is  identified  by  one  stakeholder. 

•  Actual  performance  is  measured  by  a  different  stakeholder. 

•  The  Gap  is  the  difference. 

Stakeholder  differences 

•  The  measures  will  not  match  so  the  definition  of  the  gap  and  its 
significance  must  be  a  discussion  in  order  to  make  a  decision  about  the 
gap. 

•  The  simulation  cannot  have  a  perfect  definition  of  the  gap  as  a  forcing 
function  on  the  simulation. 

Representing  the  Decision 

•  Is  actually  important  to  the  overall  process  definition. 

•  Delaying  these  decisions  may  costly  in  many  ways. 
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Customizing  the  Model 


Easy 

•  Selecting  results  to  show,  Changing  reporting  charts 

•  Running  new  scenarios 

•  New  formulas  from  stocks  or  flows  for  reporting  only 
Moderate 

•  Re-calibrating  with  new  data 

•  Tweaking  timescale  and  some  formulas 
Difficult 

•  Re-definition  of  measure  of  stock  or  flow 

•  New  feedback 

•  New  stocks  and  flows 

•  New  timescales 
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